Papers with multimodal benchmarks
MultiSkill: Evaluating Large Multimodal Models for Fine-grained Alignment Skills (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation settings for large multimodal models focus on coarse-grained evaluation without considering skill composition required by specific instructions. |
| Approach: | They propose an evaluation protocol that assesses large multimodal models across multiple fine-grained skills for alignment with human values. |
| Outcome: | The proposed evaluation protocol decomposes coarse-level scoring to fine-grained skill set-level score tailored to each instruction. |
Multimodal Evaluation of Russian-language Architectures (2026.eacl-long)
Copied to clipboard
Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria Tikhonova, Alexander Kharitonov, Yulia Lyakh, Petr Surovtsev, Denis Shevelev, Vildan Saburov, Vasily Konovalov, Elisei Rykov, Ivan Sviridov, Amina Miftakhova, Ilseyar Alimova, Alexander Panchenko, Alexander Kapitanov, Alena Fenogenova
| Challenge: | Multimodal large language models (MLLMs) are at the center of research attention, yet intelligence, limitations, and risks remain insufficiently understood. |
| Approach: | They propose an open multimodal evaluation framework for Russian-spoken architectures . the framework is instruction-based and includes 18 newly constructed evaluation tasks . |
| Outcome: | The proposed framework provides a replicable methodology for constructing multimodal benchmarks in Russian-spoken architectures. |
RetentiveKV: State-Space Memory for Uncertainty-Aware Multimodal KV Cache Eviction (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evicting KV pairs rely on the "persistence of importance" hypothesis . visual tokens display "deferred importance" but become pivotal during later decoding, authors say . |
| Approach: | They propose an entropy-driven method that reformulates KV eviction from "discrete context truncation" to "continuous memory evolution" they propose to prune visual tokens with "deferred importance" visual token exhibiting low salience but becoming pivotal during later decoding . |
| Outcome: | The proposed method achieves 5.0 KV cache compression and 1.5 decoding acceleration. |
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for expert parallelism inference suffer from a significant efficiency bottleneck . existing methods fail to address information heterogeneity and modality dynamics . |
| Approach: | They propose a training-free inference framework that scales experts without training . they propose an Entropy-Weighted Load mechanism to quantify the semantic value of visual tokens . |
| Outcome: | Experiments show that MACS outperforms existing methods on multimodal benchmarks. |
MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work suggests a prefill-stage KV cache selection method to estimate KV importance from prefilling statistics. |
| Approach: | They propose a training-free, decode-aware and strictly prefill-only KV selection method that retains key-value caching for decoding . |
| Outcome: | The proposed method outperforms existing methods under tight cache budgets on multimodal benchmarks. |
Lost in Translation: Do LVLM Judges Generalize Across Languages? (2026.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Mir Tafseer Nayeem, Amran Bhuiyan, Mizanur Rahman, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | MM-JudgeBench is the first large-scale benchmark for multilingual and multimodal judge model evaluation. |
| Approach: | They propose a multilingual benchmark for multilingual and multimodal judge model evaluation that includes over 60K pairwise preference instances spanning 25 typologically diverse languages. |
| Outcome: | The proposed benchmark includes over 60K pairwise preference instances spanning 25 languages. |